## Context
@products/tasks/evals/golden_prs/scoring.py#L88-91

<issue_description>
`_bounded` keeps only the first 120,000 characters. If important changes occur later in a large diff, the judge cannot consider them even though it must score the full behavior. This can produce misleading scores for large PRs or candidate diffs.
</issue_description>

<issue_validation>
- **Checked:** Traced `_bounded` in `products/tasks/evals/golden_prs/scoring.py` and its calls from `judge()`. Checked how `evaluate()` uses the verdict and what the report displays.
- **Found:** `products/tasks/evals/golden_prs/scoring.py:88-91` returns only the first 120,000 characters and adds a truncation marker. `products/tasks/evals/golden_prs/scoring.py:113-114` applies this independently to both diffs. The judge prompt at `products/tasks/evals/golden_prs/scoring.py:22-29` asks for a behavioral score, but does not tell the judge to treat omitted content as unknown. `products/tasks/evals/golden_prs/__main__.py:51-65` stores that score, which the report presents.
- **Impact:** Large diffs can contain relevant behavior after the cutoff, and no code ensures the retained prefix represents the full change. The judge can therefore assign a misleading score based on incomplete evidence. This meets the correctness bar for the eval's reported judge score.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Bound diffs in a way that represents changes across the full diff, such as selecting hunks across files and listing omitted files. Tell the judge what was omitted so it does not treat the partial diff as complete.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/scoring.py#L104

<issue_description>
If the Anthropic request fails after its retries, `judge()` raises and aborts the evaluation. The runner writes the candidate diff, agent log, and deterministic scores only after `judge()` returns, so a transient API failure loses the completed agent run and prevents later PRs in the batch from running.
</issue_description>

<issue_validation>
- **Checked:** Traced the judge call through `evaluate()` and the CLI loop in `products/tasks/evals/golden_prs/__main__.py`. Checked the GitHub workflow’s matrix failure behavior and artifact upload condition.
- **Found:** `products/tasks/evals/golden_prs/scoring.py:104` makes the API request without handling request errors. `products/tasks/evals/golden_prs/__main__.py:51` calls `judge()` before returning the case result; `products/tasks/evals/golden_prs/__main__.py:136-137` writes the diff, log, and scores only after `evaluate()` returns. A CLI batch therefore stops at a failed judge request without saving that case. In the workflow, `fail-fast: false` allows other matrix jobs to continue, but the failed case still has no result files for the unconditional artifact upload to preserve.
- **Impact:** A judge request failure after SDK retries can discard a completed agent run and its deterministic scores, requiring an expensive rerun to recover them. The later-PR impact applies to CLI batches, not the GitHub matrix.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Handle judge failures as a separately recorded outcome. Persist the candidate diff, agent log, and deterministic scores even when no verdict is available, and report the judge error without treating it as a genuine zero score.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/agents.py#L59-67

<issue_description>
When the timeout expires, `subprocess.run` terminates the CLI process but does not terminate its child processes. Agent CLIs can start shell commands and other tools, so those processes may keep using resources or modifying the checkout after the run times out. This can also make the collected candidate diff unreliable.
</issue_description>

<issue_validation>
- **Checked:** Reviewed the CLI launch flags and the timeout handling in `run_agent()`.
- **Found:** `run_agent()` starts the CLI with `subprocess.run(..., timeout=timeout_seconds)` at `products/tasks/evals/golden_prs/agents.py:59-67`. The CLI runs with approval and sandbox checks bypassed at `products/tasks/evals/golden_prs/agents.py:30-33`, so it can launch shell commands and other child processes. The timeout handler records the run as timed out but does not manage a process group at `products/tasks/evals/golden_prs/agents.py:67-67`.
- **Impact:** If a child process survives termination of the CLI, it can keep consuming resources or modify the checkout while `evaluate()` collects the candidate diff. This can leave local runs with lingering processes and make results unreliable.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Run the CLI in a new process session and, on timeout, terminate its process group with a graceful signal followed by a forced kill if needed. This keeps the timeout bounded for the full agent process tree.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/agents.py#L59-65

<issue_description>
`capture_output=True` buffers all stdout and stderr in memory until the agent exits. A run can last 30 minutes, and Codex emits a JSON event stream, so a verbose run can consume substantial memory and fail the evaluation worker before results are written.
</issue_description>

<issue_validation>
- **Checked:** Traced how `run_agent()` captures output and how `evaluate()` stores it.
- **Found:** `subprocess.run()` captures all stdout and stderr in memory at `products/tasks/evals/golden_prs/agents.py:59-68`, and `AgentRun` retains both strings at `products/tasks/evals/golden_prs/agents.py:73-81`. `evaluate()` combines them into the agent log at `products/tasks/evals/golden_prs/__main__.py:71`, then writes the log only after the run at `products/tasks/evals/golden_prs/__main__.py:74-78`. There is no output-size limit or incremental write.
- **Impact:** If an agent command produces large output during a run, the runner holds that output in memory until the run finishes and can exhaust available memory before it writes the results.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Stream output to a log file or process it incrementally, and keep only a bounded amount in memory. Preserve the Claude usage fields needed for scoring while avoiding full-output buffering.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/agents.py#L53-56

<issue_description>
`agent_usage` returns no usage data for Codex. The report then displays a missing cost as `$0.00`, which makes an unknown cost look measured and can mislead users comparing runtimes.
</issue_description>

<issue_validation>
- **Checked:** Traced `agent_usage` through `evaluate` into `report`, and checked the report test for missing usage.
- **Found:** `products/tasks/evals/golden_prs/agents.py:44-56` returns no usage for Codex. `products/tasks/evals/golden_prs/__main__.py:74` stores that result, and `products/tasks/evals/golden_prs/__main__.py:100` formats a missing `total_cost_usd` as `0.00`. `products/tasks/evals/golden_prs/test_golden_prs.py:188` confirms the zero-cost display for empty usage.
- **Impact:** The report presents an unmeasured cost as a measured zero, making cost comparisons between runtimes misleading.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Parse Codex usage data when available, or display an unavailable marker instead of defaulting a missing cost to zero.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/scoring.py#L61

<issue_description>
`strip()` removes leading indentation before the line is scored. In Python, indentation can change behavior or make code invalid, but lines such as `    return value` and `return value` then count as an exact match. This can inflate the deterministic line F1 score for an incorrect candidate.
</issue_description>

<issue_validation>
- **Checked:** Traced `added_lines` into `_f1` and the displayed `added_line_f1` score. Checked the existing test for added-line normalization.
- **Found:** `products/tasks/evals/golden_prs/scoring.py:61` strips leading and trailing whitespace before storing each added line. `products/tasks/evals/golden_prs/scoring.py:67-73` then counts normalized lines as exact overlap. `products/tasks/evals/golden_prs/__main__.py:98` displays that score. The test at `products/tasks/evals/golden_prs/test_golden_prs.py:84-86` covers blank-line skipping but does not check indentation.
- **Impact:** Python indentation can change syntax and control flow. The F1 score can count differently indented lines as identical, so it can overstate line overlap for a materially different candidate.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Preserve leading whitespace when extracting added lines. Skip blank lines with a separate `content.strip()` check, and only remove trailing whitespace if that is the intended normalization.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/__main__.py#L58

<issue_description>
If the Anthropic request in `judge()` fails after its retries, `evaluate()` raises before returning the candidate diff and agent log. The caller therefore never reaches `write_result()`, and the workflow's always-run artifact upload has no completed agent output to preserve. This discards work already spent running the agent and prevents retrying only the judge.
</issue_description>

<issue_validation>
- **Checked:** Traced `evaluate()`, `judge()`, the result-writing call, and the workflow’s artifact upload step.
- **Found:** `evaluate()` calls `verdict_for()` before it returns the result, diff, and log at `products/tasks/evals/golden_prs/__main__.py:55-58`. `judge()` makes the Anthropic request without handling request errors at `products/tasks/evals/golden_prs/scoring.py:103-119`. `main()` calls `write_result()` only after `evaluate()` returns at `products/tasks/evals/golden_prs/__main__.py:143-144`. The workflow upload runs even after a failure, but uploads only files already present in the results directory at `.github/workflows/golden-pr-evals.yml:166-172`.
- **Impact:** If the judge request raises, the completed agent diff and log are not persisted. The workflow cannot preserve those artifacts, and the judge cannot be retried without running the agent again.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Persist the candidate diff and agent log before calling the judge, or catch judge failures and write a result that records the judge error while retaining the deterministic scores and agent artifacts.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/scoring.py#L60

<issue_description>
This condition skips every added diff line that starts with `+++`, not only the `+++ b/...` file header. A valid source line such as `++counter` is therefore omitted from the added-line score, which can distort the reported F1.
</issue_description>

<issue_validation>
- **Checked:** Traced `added_lines` and searched the repository for valid source lines that begin with `++` after the diff prefix.
- **Found:** `products/tasks/evals/golden_prs/scoring.py:60` excludes every added line whose diff text starts with `+++`. The repository contains valid TypeScript lines such as `++index` in `frontend/src/scenes/terminal/terminalHogql.ts:150`. Another diff parser explicitly preserves content like `++i;` in `products/desktop/packages/agent/src/adapters/codex-app-server/mapping.ts:162`.
- **Impact:** When such a line appears in a candidate or golden diff, `added_lines` omits it and can distort the reported `added_line_f1` score. The trigger is uncommon, so the reported `consider` priority is appropriate.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Exclude only the actual `+++ b/...` file header, or track whether the parser is inside a hunk before counting added lines.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/scoring.py#L16

<issue_description>
`ARTIFACT_SUFFIXES` does not include `.svg`, so `is_artifact()` counts SVG assets as changed source files and `added_lines()` scores their markup. When a golden PR changes an SVG, the file recall and line F1 can penalize an agent for not reproducing an image even though the evaluation says to ignore images.
</issue_description>

<issue_validation>
- **Checked:** Traced `is_artifact()` through `changed_files()` and `added_lines()` in `products/tasks/evals/golden_prs/scoring.py:49`, `products/tasks/evals/golden_prs/scoring.py:53`, and `products/tasks/evals/golden_prs/scoring.py:57`. Checked the documented image-exclusion behavior in `products/tasks/evals/golden_prs/README.md:20` and the current golden PR files.
- **Found:** `ARTIFACT_SUFFIXES` at `products/tasks/evals/golden_prs/scoring.py:16` omits `.svg`. Both deterministic scoring paths use `is_artifact()`, so an SVG diff contributes to file overlap and added-line F1. The current golden PRs do not include SVG files, but the documented workflow supports adding more cases.
- **Impact:** Adding a golden case that changes an SVG would make the deterministic scores count image markup, contrary to the documented image-exclusion behavior.
- **Priority:** Lowered to `consider` because no current golden case is affected; the scoring mismatch applies when an SVG-bearing case is added.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Add `.svg` and any other supported image formats to `ARTIFACT_SUFFIXES`, and cover the exclusion in the scoring tests.
</potential_solution>

## Context
@products/tasks/evals/golden_prs/__main__.py#L43-45

<issue_description>
When an agent exits unsuccessfully after changing files, this code still asks the judge to score its diff. The report does not show the non-zero exit, so it presents that score as a normal completed run. When the failed agent changed nothing, the verdict is zero and the report includes it in the mean. Both outcomes can misrepresent an agent failure as its evaluation score.
</issue_description>

<issue_validation>
- **Checked:** Traced failure handling in `products/tasks/evals/golden_prs/__main__.py` and `products/tasks/evals/golden_prs/agents.py`, including how `report()` formats rows and calculates means.
- **Found:** `verdict_for()` calls the judge for any non-empty candidate, even when `agent_failure(run)` returns a failure at `products/tasks/evals/golden_prs/__main__.py:41-45`. `CaseResult` records `exit_code` at `products/tasks/evals/golden_prs/__main__.py:67-72`, but `report()` does not display it; it marks timeouts only at `products/tasks/evals/golden_prs/__main__.py:95-100`.
- **Found:** The report calculates its file, line, and judge means across every result at `products/tasks/evals/golden_prs/__main__.py:103-105`. A failed run with no candidate receives a zero verdict at `products/tasks/evals/golden_prs/__main__.py:42-44` and remains in those means.
- **Impact:** The summary does not distinguish failed runs from completed evaluations, so readers can mistake partial or failed attempts for comparable scores. Including failed runs in the means can distort comparisons between agents.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Record and display the agent failure as a separate run status. Do not include failed runs as ordinary judge scores in the report mean; if you keep a score for a partial diff, label it clearly as a failed run.
</potential_solution>